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ABSTRACT 

Objective: Implementation of a surgical checklist 
depends on many organisational factors and on socio- 
cultural patterns. The objective of this study was to 
identify barriers to effective implementation of 
a surgical checklist and to develop a best use strategy. 
Setting: 18 cancer centres in France. 
Design: The authors first assessed use compliance and 
completeness rates of the surgical checklist on 
a random sample of 80 surgical procedures performed 
under general or loco-regional anaesthesia in each of 
the 18 centres. They then developed a typology of the 
organisational and cultural barriers to effective 
checklist implementation and defined each barrier's 
contents using data from collective and semi- 
structured individual interviews of key staff, the results 
of an email questionnaire sent to the 18 centres, and 
direct observations over 20 h in two centres. 
Results: The study consisted of 1440 surgical 
procedures, 1299 checklists, and 28 578 items. The 
mean compliance rate was 90.2% (0, 100). The mean 
completion rate was 61% (0, 84). 11 barriers to 
effective checklist implementation were identified. 
Their incidence varied widely across centres. The main 
barriers were duplication of items within existing 
checklists (16/18 centres), poor communication 
between surgeon and anaesthetist (10/18), time spent 
completing the checklist for no perceived benefit, and 
lack of understanding and timing of item checks (9/ 
18), ambiguity (8/18), unaccounted risks (7/18) and 
a time-honoured hierarchy (6/18). 
Conclusions: Several of the barriers to the successful 
implementation of the surgical checklist depended on 
organisational and cultural factors within each centre. 
The authors propose a strategy for change for 
checklist design, use and assessment, which could be 
used to construct a feedback loop for local team 
organisation and national initiatives. 



INTRODUCTION 

Since the development of checklists for use in 
operating rooms by the WHO in 2008, their 



use has become increasingly widespread and 
seems to be associated with a significant 
decrease in postoperative complications and 
mortality rates. 14 Recently, however, ques- 
tions have arisen about their ease of intro- 
duction into workflow patterns and their true 
impact on safety. 5 6 These aspects require 
both high physician commitment and an 
ability to capture the work organisation. 7 8 
Thus, even if surgical checklists are widely 
used, proactive management of the organisa- 
tional changes required is indispensable for 
use to be effective and enduring. 

In January 2010, the French National 
Authority for Health (Haute Autorite de 
Sante, HAS), in collaboration with profes- 
sional colleges, introduced a modified 
version of the WHO surgical checklist for 
which it provided training sessions, written 
materials and videos. 9 This mandatory 
checklist has 22 items covering three time- 
frames: before anaesthesia ('sign in') (nine 
items), before skin incision ('time out') 
(eight items), and before leaving the oper- 
ating room ('sign out') (five items) 
(figure 1). However, at a national meeting 
6 months after the checklist was introduced, 
a representative of the French Society of 
Anaesthesia and Intensive Care declared 
that 'bureaucratic use' of the checklist was 
of no benefit and might even have a nega- 
tive impact, and that health professionals 
should take organisational factors into 
account. 

The objective of this study was threefold: to 
assess the use compliance and completeness 
rates of surgical checklists in cancer centres; 
to identify and compare barriers to effective 
use in these centres; and to develop a strategy 
for effective use. A barrier was defined as 
a work situation causing problems involving 
organisational rules, human skills, cultural 
features and beliefs. 
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Patient identification 

Portent label 
or surname, Jim mime, date of birth 



L 



SURGICAL SAFETY CHECKLIST 

Edition 2010 -01 



has 

HAUTI Al/tU«irr M JANtt 



Operating room: 

Date or procedure: 

Time of procedure (start): . 



Participating surgeon: 

Participating anaesthetist: 
Checklist coordinator: . . . 



BEFORE INDUCTION OF ANAESTHESIA 

Break before anaesthesia 



BEFORE SURGICAL PROCEDURE 

Break before incision 



AFTER PROCEDURE 

Before patient leaves operating room 



I Patient identity: 

- patient has staled his/her name 
/ailing this, other method of 
checking patient identity 



□ Yes 



Procedure and operation site arc 

confirmed 

- ideally by the patient, and in all 
cases by the patient Hie or 
specific procedure 

- any required clinical and 
paractinical information is 
available in operating room 



"Final" cross-checking within the 
team 

- patient identity correct 

- planned procedure confirmed 

- operation site identified 

- patient position correct 

- required documents available 



□ Yes 


□ No 


□ Yes 


□ No 


□ Yes 


□ No 


□ Yes 


□ No 


□ Yes 


□ N/A 



□ Yes 

□ Yes 



□ No 

□ No 



3 The patient's posilkinning is 
known by the team in the 
operating room, is consistent with 
the site/procedure and does not 
endanger the patient 



□ Yes □ N/A 



4 Equipment required during 
procedure is checked 

- surgical equipment 

- anaesthesia equipment 



□ Yes 

□ Yes 



□ No 

□ No 



Sharing within the team of essential 
information concerning 
risks/critical steps in the procedure 

- surgical 

(operation duration, specific aspects 
of this procedure etc) 



- anaesthetics 

(potential risks connected to 
patient's background or any 
treatment performed) 



□ Yes □ No 



□ Yes □ No 



Cross-checking by the team of 
critical steps and measures to be 
taken 

- patient allergy □ Yes 

- risks of inhalation, difficulties of 
intubation or mask ventilation □ Yes 

- significant risk of bleeding □ Yes 



8 Antibiotic prophylaxis given 



□ Yes □ No 
□ N/R 



Verbal confirmation by team 
members: 

- procedure recorded □ Yes □ No 

- sponges, needles, instruments 

are correctly counted □ Yes □ N/A 

- that samples and surgically 

excised material is correctly n Yes □ N/A 

labelled 

- that any equipment □ Yes O No 
malfunction and adverse events 

arc described 



1 0 Instructions for immediate post- 
operative management are 
drawn up jointly 



□ Yes 



□ NoD 
N/A 



If there is any departure from the checklist, state 
the decision that was made 



□ No 

□ No 



Abbreviations used: CI: Checklist - N/A: Sot Applicable S/R Not Recommended 



This checklist may not be modified, but may be subject to specific additional developments 

Adapted from the World Health Organization s " WHO Surgical Safety Checklist ", nmr/zWumr. who int/oatienuafeH/iafesuroerv/en. O World Health Organization 2008 

Figure 1 Surgical safety checklist published by the French National Authority for Health (HAS) (January 2010 version*). An 
updated version was published in January 201 1 . 



METHODS 
Setting 

The study was initiated in October 2009 by the French 
National Federation of Cancer Centres (FNCLCC), in 
collaboration with a research team specialising in 
healthcare management (COMPAQH, Coordination for 
Measuring Performance and Assuring Quality of Hospi- 
tals, Institut Gustave Roussy). It included 18 (out of 20) 
centres which were implementing the mandatory 
checklist. These 18 centres performed 40 620 surgical 
procedures in 2009. 

Measurement of use compliance and checklist 
completeness 

The study included measurements of surgical procedures 
performed under general or loco-regional anaesthesia 
between 11 and 29 January 2010. Cases of topical anaes- 
thesia, interventional radiology, gastro-intestinal endoscopy 
and central venous catheter implantation were excluded. 

Checklists were kept as part of each patient's medical 
record. However, as medical records were not completely 



computerised, checklist data could not be extracted 
automatically. A random sample of 80 records from each 
centre was analysed. This sample size was also based on 
published studies. 10 11 Individual scores were calculated 
for each sample and the 18 participating centres were 
ranked into three categories (top, middle and bottom). 
Definition of the categories was based on the degree to 
which the 95% CIs did or did not overlap the mean 
score. The method used partly features the Hospital 
Report Research Collaborative method. This method is 
fairer than simple ranking and accounts for uncertainty 
in the scores. 12 

The method used in this study had three categories 
and one CI — the 95% CI, whereas the Hospital Report 
Research Collaborative method had two CIs — 90% and 
99% CIs. For each centre, the compliance rate was the 
percentage of checklists found in the 80 patients' 
medical records. The mean compliance rate was calcu- 
lated as the sum of the compliance rates for each centre 
divided by the number of centres. 

For each centre, the completeness rate was the 
percentage of complete checklists found in the 80 
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patients' medical records. A complete checklist is 
a checklist in which all items have been ticked. The 
mean completeness rate was calculated as the sum of the 
completeness rates for each centre divided by the 
number of centres. 

Analysis of barriers to effective checklist use 

To identify barriers to effective checklist use and for 
good internal validity, data were collected in three ways 
(collective and individual interviews, an email ques- 
tionnaire sent to surgical staff and direct observations), 
and the results triangulated. First, two researchers (AF 
and EM) collectively interviewed voluntary staff in 16 of 
the 18 centres about the benefits and drawbacks of 
checklist implementation during a 3 h session. Four 
surgeons, three anaesthetists, two nurses, six senior 
nurses and one quality manager took part in this 
collective interview. Then, eight other key surgical staff 
from different centres (two surgeons, three anaesthe- 
tists and three senior nurses) were scheduled for indi- 
vidual interviews using as substrate a semi-structured 
questionnaire (25 questions) based on the results of the 
collective interview. A ninth person, a surgeon, 
declined the interview because of a full agenda. All 
eight individual interviews were conducted by the same 
researcher (AF) to ensure that a similar emphasis was 
given to each question in each interview and that the 
meaning and importance ascribed to each barrier was 
explicit. The interviews lasted 30 min on average. 
Second, to capture a collective perspective, a contact 
person in each centre was sent an email questionnaire 
consisting of 25 questions plus five questions on further 
barriers brought up by the interviewees. This email 
questionnaire was piloted for clarity at the Institut 
Gustave Roussy and took about 10 min to complete. All 
centres returned the questionnaire. In most cases, it was 
completed by operating room staff but sometimes by 
quality department staff. Lastly, a researcher (AF or 
EM) made direct observations on all the barriers that 
had cropped up during the interviews and in the 
completed questionnaires. These observations were 
made over 20 h in two centres, one with a low (52%) 
and one with a high (84%) proportion of complete 
checklists. 

Two researchers (AF and EM) processed and 
compared the data from the interviews, email question- 
naire, and direct observations. Data were categorised as 
follows. Axial coding was used to relate responses and 
create a typology of the barriers hindering effective 
checklist use. A first typology was created after the 
collective interview. It was revised twice, first after addi- 
tion of the data from the individual interviews and then 
after addition of the data from the questionnaires. 
Vertical coding was used to better define the content of 



each barrier. Dependency chains were sought to distin- 
guish data related to missing checklist items and data 
related to barriers. 13 The content of each barrier was 
defined after the interviews and fine-tuned after the 
return of the questionnaires and after making direct 
observations, using the revised typology. The typology 
and barrier contents were discussed and validated by the 
staff who had participated in the collective interview, on 
the basis of their working experience, during a second 3 
h session during which discrepancies were resolved and 
minor adjustments were made. 

RESULTS 

Compliance of use and checklist completeness 

The analysis included 1440 surgical procedures (80 per 
centre). Overall, 1299 checklists were handed in and 
28 578 items were analysed. The mean compliance rate 
was 90.2% (range 0—100) and the mean completeness 
rate was 61% (range 0—84) (figure 2). The 0% score was 
allocated to a centre unable to retrieve proof that the 
medical reports had been checked. The median 
compliance rate was 98.75 and the median completeness 
rate was 66.25. For the first quartile, the compliance rate 
was 92.50 and the completeness rate was 52.81. The 
values for the third quartile were 99.69 and 79.38, 
respectively. Most missing items (47%) occurred during 
'sign out' (table 1). 

Barrier typology 

Eleven barriers to effective checklist implementation 
were identified. Their frequency, as judged from the 
answers to the email questionnaire, is given in 
decreasing rank order, with an example of content taken 
from the individual and collective interviews and/or 
from direct observations (table 2). Other elements of 
content support some key aspects of each barrier. 

The most common barrier (16 centres) was duplica- 
tion with existing processes that already covered several 
of the items in the surgical checklist. For example, 
checking patient identity, accounting for sponges or 
adverse events reporting were already carried out before 
the use of the checklist, as procedures and documents 
existed for those examples. For adverse events reporting, 
an electronic system was also used in most of the centres. 
Therefore there was duplication in documents for 
reporting these events. 

The next most common barrier (10 centres) was lack 
of communication between the surgeon and the anaes- 
thetist at the end of the surgical procedure. The surgeon 
might leave the centre before 'sign out' and the anaes- 
thetist might return during recovery after skin closure. 
This jeopardised the sharing of information on patient 
management. 
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Figure 2 Rates of compliance of 
use (A) and completeness (B) 
recorded for the surgical checklist. 
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Staff in nine centres found that the checklist took too 
long to complete as they already had a heavy workload, 
and did not perceive the added benefit. According to 
one surgeon, 'the checklist could generate delays and 
conflicts'. High staff turnover, especially of nurses, in 
nine centres was also considered an obstacle to checklist 
implementation despite provision of training. In nine 
centres, some items did not make sense because they did 
not fit in with customary operating room practices (eg, 
needle counts after disposal) or because their timing was 
inappropriate. 

The binary (yes/no) response system was ambiguous 
and confusing according to eight centres, that is, could 
a non-'yes' answer prevent moving on to the next ques- 
tion and postpone surgery? Seven centres reported risks 
that were not prevented by application of the checklist, 
even if it were fully completed (for example, thrombo- 
embolism, contamination and viral risk (HIV, HCV, 



HBV)). Staff in six centres had communication prob- 
lems during checks. According to the nursing staff 
completing the checklist, the surgeons and anaesthetists 
who were ultimately responsible for the surgical proce- 
dure did not always listen to the items when they were 
read out. Nurses were therefore concerned about the 
legal implications of signing the checklist as they might 
be held accountable for errors. In five centres, staff 
thought that repeating questions that had already been 
answered several times might generate anxiety in 
patients about to undergo anaesthesia but this was not 
supported by direct observation. Five centres reported 
that items could be ticked off even when items were not 
checked because of time constraints. Items were ticked 
only to comply with the management audit; therefore, in 
this situation, checklists failed to improve patient safety. 

However, during the 20 h of observation in two 
hospitals, many items were left unchecked or answers 
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Table 1 Frequency of missing items 



Item 



Number of 
times missing 



Min (%) 



Max (%) 



Sign in 



"1 Pationt iHontitw 
I. i cUlfcMIl lUcMIULy 


7 


U.04 


n 


2. Procedure and operation site confirmed 


12 


0.93 


0 


3. Clinical and paraclinical information required 


74 


5.71 


0 
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d. Mnaesinesia equipment requirea 
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mask ventilation 








y. oiymircani nsK ot Dieeamy 
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Time in (cross-checking) 




29.45 




I U. rdLlfcMIl lUollLlly UUIIfcJUl 


^n 
ou 


o "31 


n 


1 1 . Planned procedure confirmed 


OO 


O 7fl 
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OO 
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U 


io. rdLieru position correct 
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Sign out (verbal confirmation) 








18. Procedure recorded 


97 


7.48 


0 


19. Instrument, swab and needle counts correct 


118 


9.10 


2 


20. Specimens and samples correctly labelled 


103 


7.94 


0 


21 . Problems with equipment and adverse events 


143 


11.03 


0 


reported 








22. Postoperative orders drawn up jointly by 


154 


11.87 


0 


surgeon and anaesthetist 








Total 


615 


47.42 




Grand total 


1297 


100.00 





2.5 
3.75 
52.5 

5 

13.75 
12.5 
5 

6.25 
41.25 



■3 75 



13.75 
13.75 
13.75 
13.75 
32.5 
18.75 
11.25 
15 



23.75 
26.25 
26.25 
33.75 

41.25 



given by the wrong person but there were no instances of 
items being ticked off without actually checking the item 
at the beginning of the surgical procedure or at the end. 

Variability between centres 

Checklist use was compared by direct observation in two 
centres with similar case mixes but with different 
compliance rates (100% and 92.5%) and completeness 
rates (84% and 52%). In the centre with poor results, 
several staff members had not been involved in checklist 
implementation. Many items were not checked. The 
head surgeon in this centre had attended the training 
session but considered 'administrative tools' unsuited to 
assessing his field of work. He was therefore against using 
the checklist. His staff thought that the measurements 
that the anaesthetist, who was working on the centre's 
information systems, had made using the checklist were 
'boring' and irrelevant to just-in-time management. 



DISCUSSION 

Mean compliance with mandatory checklist use was high 
(90.2%) at introduction but far short of the desired 
100% score. The mean percentage of complete check- 
lists was relatively low (61%; range 0—84%), indicating 
that hospital staff encountered problems with imple- 
mentation after checklist adoption. These results 
support those for the SURPASS checklist for which the 
completion rate was about 80% for most items but less 
than 30% for others. 2 

Eleven organisational barriers to implementation were 
identified and several of those already identified, such as 
poor communication between the anaesthetist and 
surgeon, lack of leadership, inappropriate timing for 
checking an item, time taken up by checklist comple- 
tion, and difficulty in identifying the role and responsi- 
bility of each staff member, were confirmed. 6 8 14 
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Table 2 Frequency of occurrence of 1 1 barriers with illustrative examples 



Barrier 



Centres 
(out of 18) 



Illustrative examples 



Duplication with existing checks 

Poor communication between 
anaesthetist and surgeon 
Time consuming 

Does not make sense 



Inappropriate timing 

Ambiguity 

Unaccounted risks 

Oral confirmation of items 
Identification of the role and 
responsibility of staff 

Patients' attitude to questions 

Gaming 



16 
10 
9 
9 



Checking patient identity, accounting for sponges 
and adverse event reporting 
Did not always use the same document to record 
postoperative orders during 'sign out' 
Checklist too long to complete, especially when very 
busy (eg, emergency surgery, end of day) 
Staff in some operating rooms are not accustomed to 
count needles and this may not even be possible after 
disposal into appropriate containers during surgery to 
avoid injury 

Difficult to check sample labelling at the end of the 
procedure if the samples were sent to the pathology 
laboratory during surgery 

Did a 'yes' response for 'allergies' mean that the patient 
had an allergy or that the risk of allergy had been checked 
Checklist did not cover skin preparation and postoperative 
prevention of pain or vomiting 
Reading out the entire list was found unnecessary 
Direct observation had difficulty in identifying the person 
implementing the checklist during emergency and/or short 
procedures as all staff were totally engrossed in their task 
Asking the patient his or her name three times over a very 
short time may cause alarm 
Ticking off unchecked items at the end of the day 



However, these results should not overly dampen 
enthusiasm and prevent use of the checklist as a 'short, 
uncluttered by noncritical items, and carefully worded' 
reminder of key actions. 8 

Two main reasons might explain the barriers identi- 
fied. The first relates to work organisation. A checklist is 
often put across as a tool to enhance communication 
and as a reminder in stressful circumstances but, like 
other operational tools, it impacts on the organisation of 
work. 8 1 Organisational changes are needed while 
implementing a surgical checklist in operating rooms, 
for example, harmonisation with existing adverse-event 
reporting systems and allocating the time needed to 
complete the checklist, communicate it, and check that 
it is complete. The second reason relates to professional 
relationships and cultural habits in clinical practice. 
Verbal communication between health professionals 
(surgeons, anaesthetists and nurses) has to be egalitarian 
for checklist use to be effective but the findings suggest 
that operating room staff practices are rooted in 
a time-honoured hierarchy, at least in France. 

The number of unchecked items varied widely across 
centres, and was probably a reflection of the usual 
behaviour of the surgical team and of the 'sense' they 
made of the checklist. 14 Operational tools uncoupled 
from operational activities because they lack 'sense' may 
fall short of the expected level of use or induce unde- 
sirable practices such as inhibition in care practice or 



gaming. 16 " 19 Any strategy for organisational changes to 
be made on checklist introduction should therefore 
take into account the cultural maturity and history of 
operating room staff. 

The findings of this study have practical implications 
for the design (areas of applicability), use (local 
contexts) and assessment of checklists by national health 
authorities. Checklist design should account for the 
organisational constraints to which the activity is subject. 
Clarity is also essential. HAS added the option 'Not 
Applicable' to the 'Yes/No' boxes of most checklist items 
in May 2010 and included all 11 barriers in their 
checklist's guideline to improve checklist use. As for all 
quality improvement measures, each centre should 
develop its own strategy for work organisation as indi- 
cated by the wide variation in checklist implementation 
across centres. 20 For example, one cancer centre laid 
down five ways of improving surgeon— anaesthetist 
communication on peroperative risks, which was one of 
the most important barriers identified. In line with 
Parand et aFs view that the priorities and concerns of 
surgeons are paramount, the surgeons and anaesthetists 
of this centre listed unexpected events, entered these 
events into a common thesaurus, set up an alert system 
for staff outside the operating room, made the presence 
of the anaesthetist compulsory at the end of each 
surgical procedure in order to share information on 
drug prescriptions, and asked for regular feedback on 
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experience. At the close of the present study, the working 
group formed by the members of staff taking part in the 
collective interview developed a formal assessment plan 
with the support of the national Cancer Centre Federa- 
tion. This plan included feedback from centres on how 
they dealt with the 1 1 barriers with a view to sharing of 
experiences, longitudinal follow-up of compliance and 
completeness, longitudinal measurement of a list of 
complications (to be drawn up in 2011) in order to 
include perioperative safety, and a yearly questionnaire 
assessing the cultural maturity of each centre. 

This study has several limitations: The announcement 
by HAS that the checklist was mandatory may have 
resulted in high staff motivation and overestimation of 
checklist use (both compliance and completeness). 
Overestimation may also have arisen from the presence 
of an external observer (Hawthorne effect). However, 
completeness may have been underestimated as, despite 
training, staff were still relatively unfamiliar with the 
checklist. The staff members taking part in the collective 
interview were a source of data but were also involved in 
the validation of the analysis performed by the 
researchers. This may have led to some bias in inter- 
preting the results, although discrepancies with the 
researchers' analysis were resolved by discussion and led 
to only minor adjustments. Lastly, situations such as 
interventional radiology and local anaesthesia were 
excluded. Whether the findings on surgery in cancer 
centres can be generalised to other types of surgery in 
other settings requires further analysis. 

In conclusion, before considering the impact of 
a surgical checklist on safety, it is necessary to consider its 
adoption by staff and to highlight the barriers to effec- 
tive use. Consideration should be given to tailoring 
checklists to services and areas with different needs (eg, 
the lab specimen labelling example) and/ or education 
about items that are new but still considered important 
by the proponents of the checklist (eg, the needles 
examples). The strategy for change described in this 
study could be used to construct a feedback loop for 
local team organisation and national initiatives. 
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